Repository navigation
[Klaud Cold] Add MiniMax-M3.1-Flash DSpark golden AL curve and SPEED-Bench collector / [Klaud Cold] 新增 MiniMax-M3.1-Flash DSpark 黄金 AL 曲线与 SPEED-Bench 采集脚本 - #4425
Conversation
SPEED-Bench coding AL for the built-in DSpark drafter, levels 1-7, thinking_on (the M3.1 template is adaptive-only), measured on B200 TP4 with the vLLM day-0 image. Give b200-nscale a writable download root so script runs can fetch unstaged checkpoints.
|
Thanks for the contribution!
中文感谢你的贡献!
|
|
Generally most InferenceX submissions don't need to add changes to the infx package. On the off chance they do, generally the diff is small. Please prompt your agent to clean your PR to keep your diff to only necessary & minimal changes too. We have an increased code quality bar for changes to the infx package. 中文大多数 InferenceX 提交通常不需要修改 infx 包。即使确实需要,改动通常也很小。请让你的 agent 清理 PR,使改动只包含必要且最小的内容。infx 包的改动有更高的代码质量要求。 |
| local bench_rc=$? | ||
| if [[ $bench_rc -ne 0 ]]; then | ||
| echo " -> benchmark client exited rc=$bench_rc (thinking=$mode dspark=$mtp); cell will be N/A" | ||
| fi |
There was a problem hiding this comment.
🟡 (optional) If the benchmark client fails partway, the collector still records an AL from partial data and the job passes, even though the log says the cell will be N/A. After bench_rc is non-zero, run_cell still reads the counter deltas. It stores al whenever delta_drf > 0, so a crash after some drafts writes a real-looking value into the golden YAML. Fix: when bench_rc is non-zero, set AL_RESULT[${mode}_${mtp}] to N/A, call cleanup_server and return before computing deltas, so the log message and the matrix agree.
Why this was flagged
Trigger: the client python3 "$BENCH_DRIVER" exits non-zero midway, for example from request timeouts, OOM or a server crash after a few batches. The script prints "cell will be N/A" at line 237 but does not return. Lines 240-249 still compute AL from whatever drafts the server counted, and line 251 stores it. The MEASURED check at the end of the file passes because that cell is not N/A. The emitted OUT_YAML then holds an AL computed from a subset of the 80 SPEED-Bench prompts, with no marker that the data is partial. That value can get committed as a golden curve, which AgentX synthetic acceptance consumes. The Kimi collectors copy the same logic, so this new file brings the defect back rather than fixing it. Nothing guards against it: N/A is only assigned when the server fails to start or when delta_drf is 0.
Verification: Line 235 sets local bench_rc=$?, and lines 236-238 only echo " -> benchmark client exited rc=$bench_rc ...; cell will be N/A". Nothing after that returns or sets N/A. Line 246 checks only if [[ "$delta_drf" -gt 0 ]], and line 252 stores AL_RESULT["${mode}_${mtp}"]="$al". The final guard at lines 293-303 fails the job only when every cell is N/A.
| if [[ -n "${MODEL_PATH:-}" ]]; then | ||
| if [[ ! -d "$MODEL_PATH" || -z "$(ls -A "$MODEL_PATH" 2>/dev/null)" ]]; then | ||
| hf download "$MODEL" --local-dir "$MODEL_PATH" | ||
| fi |
There was a problem hiding this comment.
🟡 (optional) If a download into the newly writable b200-nscale model root is interrupted, every later collection skips it and fails all cells until someone deletes the folder. Line 94 only downloads when MODEL_PATH is missing or empty. A partial hf download --local-dir leaves the folder non-empty, and the exit status is never checked. Fix: on every run, call hf download "$MODEL" --local-dir "$MODEL_PATH" (it resumes and skips finished files) and exit if it fails. Other alternative: write a completion marker after a successful download and check for that marker instead of a non-empty folder.
Why this was flagged
Before this change, b200-nscale had no models.download-root. resolve_model in infx/launch/drivers/script.py:47-51 raised LaunchError for unstaged checkpoints, so nothing was downloaded there. runners.yaml:452 now resolves MODEL_PATH to /models/MiniMax-M3.1-Flash on the shared /data/home/sa-shared/gharunners/models volume, and that folder outlives the job. If a run is cancelled, hits SALLOC_TIME_LIMIT, or hf download fails partway (quota, network, gated token), the folder already holds files such as .cache/huggingface/. The test at line 94, -z "$(ls -A "$MODEL_PATH")", is then false on every later run, so the download is never resumed. The script has no set -e (line 25) and ignores the download's exit status. It goes on to start seven vllm serve processes against incomplete weights, every cell is N/A, and line 299 exits 1. This repeats until someone cleans up the shared folder by hand. Other download paths in this repo are completeness-checked and flock-serialized (perf-changelog.yaml:8042). This one has neither, and nothing else on this path checks the weights.
Verification: Trigger: a collection on b200-nscale is cancelled or has hf download fail partway. The script only downloads when the folder is missing or empty (minimaxm31flash_fp4_b300_vllm.sh ~lines 94-96). Nothing checks the exit status of hf download. After an interruption ls -A therefore returns something, and every later run skips the download.
| # AL is concurrency-independent (per-token accept/reject; no spec-disable-by-batch is | ||
| # set), so batch the SPEED-Bench pass to stay under the CI wall-time limit; conc=1 | ||
| # blew the 8h budget on an earlier Kimi model. |
There was a problem hiding this comment.
🟡 nit (optional): the new script keeps comment types AGENTS.md bans, a file header and a history note. sweep:(earlier Kimi model|^# MiniMax-M3.1-Flash vLLM SPEED-Bench AL matrix collector). Lines 3-23 are a header describing the file. Lines 38-40 include "conc=1 blew the 8h budget on an earlier Kimi model", which AGENTS.md cites as a banned past-results comment. Fix: delete the header block and keep only the technical reason on lines 38-39 (AL is concurrency-independent).
Why this was flagged
AGENTS.md:64 says "No header comments" and forbids describing what the file is at its top. AGENTS.md:65 says "No history" and forbids comments about past results. Lines 3-23 of the new file describe the collector, the recipe it follows and the filename convention. Lines 39-40 cite a past Kimi run blowing the 8h budget. The base has no such file. These comments were copied from the Kimi collector and now land in a new file.
Verification: nit. Trigger: the file is new in this PR, so the comments come in with this change. Lines 3-23 of inferencex-e2e/benchmarks/single_node/speedbench/minimaxm31flash_fp4_b300_vllm.sh are a top-of-file header. Lines 39-40 say "conc=1 / blew the 8h budget on an earlier Kimi model.", which is a comment about a past result. AGENTS.md:64 and AGENTS.md:65 ban both.
| ("dsv4dsparkprob", {"method": "dspark"}, "dsv4-pro-0813-dspark"), | ||
| ("dsv41flash", {"method": "dspark"}, "dsv41flash_dspark"), | ||
| ("musespark", {"method": "dflash"}, "musespark_dflash"), | ||
| ("minimaxm31flash", {"method": "dspark"}, "minimaxm31flash_dspark"), |
There was a problem hiding this comment.
🟡 nit (optional): the new test row adds no coverage and breaks the repo's test rules. It hits the default f"{model}_{method}" branch of curve_name (curves.py:47), which the dsv41flash and musespark rows already cover. Its is_file() assertion only checks that a committed data file exists. Fix: drop the minimaxm31flash row. test_every_committed_curve_is_valid already loads and range-checks the new YAML.
Why this was flagged
curve_name in infx/golden_al_distribution/curves.py:41-62 has special cases only for dsv4*, minimaxm3 and kimik3. For minimaxm31flash + dspark it returns the plain f"{model}_{method}" from line 47. The existing rows ("dsv41flash", {"method": "dspark"}) and ("musespark", {"method": "dflash"}) already reach that branch. AGENTS.md "Test quality" item 9 forbids tests that duplicate a covered path. Item 4 forbids pinning checked-in data, and (GOLDEN_DIR / f"{expected}.yaml").is_file() does exactly that. The new curve is already validated through real code by test_every_committed_curve_is_valid, which globs every YAML in GOLDEN_DIR. So the row only adds a test that fails when someone renames a data file, not when behavior regresses.
Verification: nit: the new parametrize row (test_golden_al_distribution.py:42) adds no branch coverage. In curves.py curve_name, minimaxm31flash with dspark skips every special case and returns the plain curve = f"{model}_{method}" (line 47), which the dsv41flash and musespark rows already reach. test_every_committed_curve_is_valid already range-checks the new YAML.
…k on the golden AL curve Split B200 into its own PR. TP4 drafts 7 DSpark tokens (vLLM day-0 GB300 AgentX frontier), DEP4 drafts 3 (recipes.vllm.ai Blackwell); throughput runs use the minimaxm31flash_dspark golden AL from #4425.
Summary
Golden SPEED-Bench acceptance-length curve for MiniMax-M3.1-Flash DSpark (
minimaxm31flash_dspark.yaml), plus the collector that produced it. AgentX DSpark recipes for this model need it for synthetic acceptance.Measured in SpeedBench AL Collection run 39895397914 on B200 (
b200-nscale), TP4, imagevllm/vllm-openai:minimax-m31-flash@sha256:86f582da…. Each level ran all 80 prompts with 0 failures.Collector (
benchmarks/single_node/speedbench/minimaxm31flash_fp4_b300_vllm.sh)minimax_m3parsers.dspark/,dspark_block_size7), so there is no draft model path and the levels go 1–7. It usesFLASH_ATTNdrafter attention, an FP8 draft KV cache, probabilistic drafting and block rejection.reasoning_effort. Unlike M3 it has nothinking_modekwarg, and the day-0 vLLMminimax_m3reasoning parser and renderer add no off switch. Collected withthinking-kwargs={}so prompts match what the AgentX server sends; AgentX always looks upthinking_on._b300_in the filename comes from thespeedbench-al.ymlnaming convention.configs/runners.yamlb200-nscalegetsdownload-root: writable-models(/data/home/sa-shared/gharunners/models, owned bysa-shared, group-writable), mirroringb300-dsxe. Before this, script-driver runs on this cluster could only use staged checkpoints.Dispatch used to produce the curve:
The B200/B300 AgentX recipes (#4421) build on this.